Taipower Smart Grid

Smart grid has been promoted by Taipower for several years. For the power operation data collection purpose, the operation technique (OT) network needs to be connected with information technique (IT) network, e.g. intranet. Additionally, the proprietary information system environment of smart grid is gradually replaced by the open system; thus the cybersecurity has become a big issue.

Apparently, penetration testing is an adequate solution for solving a complicated information system, e.g. smart grid, cybersecurity problem. Taipower has therefore planned to develop an eﬀective penetration testing platform for ﬁnding out the cybersecurity vulnerabilities of smart grid.

Refer to https://lp3.com/tips/5-beneﬁts-of-penetration-testing web pages, numerous beneﬁts are founded for employing penetration testing.

6-1-1　Detect and Arrange Security Threats

A penetration testing estimates the ability of an organization to defend its applications, networks, users and endpoints from internal and external attempts to dodge its security controls to achieve privileged or unapproved access to protected assets. Penetration testing results conﬁrm the threat posed by particular security vulnerabilities or faulty processes, allowing IT management and security experts to arrange remediation eﬀorts. Organizations can more eﬃciently anticipate emergent security threats and avoid unauthorized access to crucial information and critical systems by executing regular and complete penetration testing.

6-1-2　Circumvent the Rate of Network Downtime

Penetration testing supports an organization to evade these ﬁnancial setbacks by proactively detecting and addressing threats before security breaches or attacks take place.

6-1-3　Protect Customer Loyalty and Company Image

Even a single occurrence of compromised customer data can destroy a company’s brand and negatively impact its bottom line. Penetration testing helps an organization avoid data incidents that may put the company’s reputation and reliability at stake.

6-1-4　Service Disturbances and Security Breaches are Expensive

Security faults and any associated disruptions in the performance of applications or services may cause debilitating ﬁnancial harm, damage an organization’s reputation, grind down customer loyalties, generate negative press, and incur unanticipated ﬁnes and penalties. Frequent employment of penetration testing avoids these expenses of the organization.

Penetration testing helps Taipower avoid smart grid invasions. It is better for Taipower’s business to proactively maintain its security than to face extreme losses, both to its brand equity and to its ﬁnancial stability.

Furthermore, penetration testing should be carried out whenever there is a change in the smart grid information infrastructure by highly experienced experts who will scrutinize internet connected systems for any weakness or disclosure of information which could be used by an attacker to compromise the conﬁdentiality, availability or integrity of the network.

In order to cope with the challenges of information security issue derived from the smart grid as well as to gain the above beneﬁts, Taipower created a research project for developing a platform for the penetration testing of smart grid information system. This project focused on IEC 61850 using the speciﬁed scenarios and aimed at understanding the current international smart grid funding through the security-related standards and regulations, and contains the standard of industrial control systems used to validate the information security functions.

Currently, The IEC 61850 standard substation automation information system is pushed forward as long asTaipower smart grid policy is decided. A typical conﬁguration of Taipower ‘s IEC 61850 substation automation system will be formed as shown in Figure8 and Figure9.

Figure 7 Taipower’s IEC 61850 substation automation information system

In Figure7, great numbers of IT devices are used as well as the TCP/IP networks. The cybersecurity must be considered in this situation. A penetration testing platform is then imported for testing the vulnerability and cybersecurity holes of Taipower’s IEC 61850 substation automation system.

Practically, the IEC 61850 substation automation system is truly installed in a physical isolation network area. The conventional cybersecurity attacks, such as DoS, do not seem to be possible happen in this system. However, along with the use of open information technology system, the corresponding risk of vulnerability and virus may be raised. Also, with the requirements of data exchange between IT and OT system, a data transfer route is possibly created. This data transfer route may result in a hacker attack. Thus the cybersecurity of the substation automation system becomes a main issue. The penetration testing, therefore, is an indispensable means for this issue.

Penetration testing is an authorized and proactive eﬀort to assess the security of an IT infrastructure by carefully running tests to exploit vulnerabilities of the system, comprises in an operating system, misconﬁgurations, service errors, and even unsafe end-user behaviors. These evaluations help conﬁrm the eﬀectiveness of defensive mechanisms and adherence of end-users to security procedures. Hence, for the sake of avoiding the hacker attacks as well as the possible vulnerability in the smart grid, a smart grid penetration testing is needed to carry out at the stage before the commercial operation.

Taipower outsourced a smart grid penetration testing platform development project to Onward Security Corporation by 2015. This project has developed a platform for testing both IEC 61850 and DNP3.0 over TCP/IP SCADA systems, refer to Figure 8 and Figure 9.

Figure 8 Penetration testing platform for IEC 61850

Figure 9 Penetration testing platform for DNP3.0 over TCP/IP

Onward Security has also proposed an analysis result for up to date cybersecurity standards that include IEC 62351, NERC CIP 002-009, NISTIR 7628, IEEE 1686 and ISO 27019. Additionally, this analysis result states which of these cybersecurity standards are adequate for Taipower applying.

For the IEC 61850 cybersecurity purpose, Onward Security provides a recommendation for the mapped portion of IEC 62351 to Taipower. This recommendation also proposes the procedures for adopting cybersecurity in IEC 61850 and DNP3.0 over TCP/IP SCADA systems.

6-2　Methodology

The content described in this section refers to two documents for industrial control system penetration testing method. They are “Cyber Security Assessments of Industrial Control Systems Good Practice Guide” issued by the United StatesDepartment of Homeland Security in 2011 and “Guide to Penetration Testing for Electric Utilities”, Version 3, issued by National Electric Sector Cybersecurity Organization Resource (NESCOR). The project for developing the penetration testing platform will evolve and analyze a tailored penetration testing procedure based on the above methodology for Taipower smart grid.

The critical information infrastructure protection (CIIP) was included in the national security policy of every country after 911 events. In 2013, Fifty percent of cybersecurity attack events, counted by DHS ICS-CERT, were focused on energy facility. In The United States, National SCADA Test Bed, NSTB, was set for resolving the intrusion and cybersecurity problems of ICS in critical infrastructure by 2003. The test bed provided the corresponding system test and developed the relative risk assessment method as well as the use cases for ICS in every application realm. The brief penetration testing methods are stated as the following sections:

6-2-1　The Penetration Testing Methods Used by the United States Power Company

1.　Security Evaluation Methods

The Department of the United States Homeland Security announced”Cyber Security Assessments of Industrial Control Systems Good Practice Guide” in 2011. This document proposes the test procedures for the assessment of ICS. The test procedure includes:

The Test Plan:

It is mutually beneﬁcial for the assessment team and the asset owner to create a test plan before testing begins so that both entities know how the assessment will operate, including the rules of engagement, attack vectors and points of contact. However, the level of eﬀort put into the test plan is a grey area that has to be decided by the asset owner. The asset owner may be more comfortable if there are a lot details included in the test plan so that this document can act as a contract with the assessment team. However, the assessment team does not need great detail in this document, the exception being the rules of engagement. In fact, it may be a hindrance to the assessment team for the test plan to include many details (discussed in the next section). Ultimately, the time and money spent creating the test plan may be subtracted from the testing operations budget. This fact could potentially restrict the assessment team from accomplishing some portion of the desired testing.
Choosing the Assessment Team:

The asset owner chooses the testing organization or provider but may have little control over the actual members of the assessment team. Information about the team members should be provided by the organization hired to perform the assessment. This information may include certiﬁcations, experience, skills and conﬁrmation of background checks. ICS cyber security assessments diﬀer signiﬁcantly from standard IT-type assessments. It is imperative that members of the assessment team have experience with assessing ICSs and are aware of the limitations and challenges associated with testing in a production environment. The asset owner should validate the team’s references to ensure that the team has adequate ICS experience. The testing organization should provide the asset owner with a methodology of how assessments are performed in a production ICS environment. The methodology should include a list of typical tools used by the team and indications of when and how the tools will be used.

The roles and responsibilities of each team member should be clearly deﬁned and communicated to the asset owner. If the assessment is to include a network analysis, at least one team member should have qualiﬁed networking experience and possibly network certiﬁcations such as those available from Cisco. At least one team member should be familiar with a number of the network protocols unique to ICS (e.g. DNP3, the Modbus suite, PROFINET, PROFIBUS, ICCP, OPC, etc.). This individual is responsible for analyzing network traﬃc and assessing the conﬁgurations of network devices such as ﬁrewalls, switches and routers. Other team members should be proﬁcient in coding, reverse-engineering, protocol analysis and exploit development. The team members should be familiar with multiple languages such as C, C++, Python, Perl and assembler. Many ICSs have non-Windows operating systems and the asset owner should ensure that team members are familiar with the operating systems used on the target system.If possible, the asset owner should request individuals familiar with the protocols, programming languages, applications and operating systems used by the ICS.

As the assessment team will have access to sensitive information, the asset owner should be provided with conﬁrmation that appropriate background checks have been performed. The control methods for the information acquired during the assessment may be deﬁned in a legal document such as a non- disclosure agreement.

Selecting the Attack Vectors:

One of the pre-assessment meeting tasks is to establish a set of initial attack vectors to include in the test plan. Many criteria may be used to select these items, but use caution in the level of detail speciﬁed. A detailed test plan prescribes exactly what to test, which will ensure that the assessment team covers the items identiﬁed. However, all the testing hours might be consumed ﬁlling in the details in the test plan without uncovering easily accessible vulnerabilities in other areas not included in the plan. An example might be that the asset owner wants to know if an attacker can take control of the front end processor (FEP) based on the communications allowed from the remote terminal unit (RTU). While this may be a valid concern in this particular installation, it might be overshadowed by the privileges extended to the ICS vendor maintenance connection. The alternative is to specify functions or transitions (discussed below) that present a potential attack vector. An example might be to test whether an attacker can make a network transition from one of the DMZ servers to a server inside the control network.

The most important part of planning a cybersecurity assessment is that the plan should not constrain the assessment team to approach a problem from only one direction. The test plan should loosely deﬁne what to test and never how to test it. This allows the cyber team to use all their skills to accomplish the goals. After all, by deﬁnition, a potential attacker is not going to follow the rules of engagement.
The general ICS assessment process overview is summarized in Figure 10 below.

Figure 10 Cyber security assessments of ICS process ﬂow chart

2.　Vulnerability Assessment (VA) and Penetration Testing

Generally, Vulnerability Assessment (VA) means analyzing the security of the target system by means of an automatic tool. This process cannot take advantage of system vulnerability. Nor detecting some cybersecurity problems, e.g. DoS.However, Penetration Testing (PT) is focused on obtaining the authorization and privilege of the target system. Several kinds of PT tool are utilized simultaneously based on the knowledge and experience of cybersecurity. Practically, it should be much careful and attendant to progress VA and PT on an ICS system.

Penetration testing should be performed on a periodic basis depending on the criticality of the targeted system. This can be performed as a broad penetration test encompassing several control systems (such as an entire testing or staging control network), a targeted penetration test with a restricted scope of a single control system (management server to its controlled devices), or to test a single component of a larger system, such as a historian or a reclosure. It is recommended that performing this type of assessment in testing or staging environments on an annual basis or after any major systems upgrades or changes to the systems in question.

Penetration tests should start with a review of the target architecture to help the testing team gain a deeper knowledge of the target system. This will help the testing team understand the intended functionality of the targeted system, its expected security posture from an architectural perspective, and the security risks that vulnerability could pose to the organization. This is best performed through interviews with knowledgeable experts from both the product’s vendor and the utility or asset owner deploying the target system.A practical process ﬂow for smart grid penetration testing planning and processing is shown as Figure 11.

Figure 11 The practice procedure for electric utilities penetration testing

All penetration tests should start with proper planning and scoping of the engagement. Once that is complete, the penetration testing tasks can be broken into the four distinct task categories displayed in Figure 11. Each of these task categories also requires diﬀerent skill sets from the testing team. If there is suﬃcient staﬀ, these four penetration task categories can be performed in parallel. Once these tasks are completed, the team should perform a gap analysis to verify all desired tests have been performed and all goals met. Finally, the team should generate a report documenting their ﬁndings, interpret these ﬁndings in the context of the utility’s deployment, and develop recommendations to resolve or mitigate these vulnerabilities.

The color diﬀerence between these four penetration task categories in Figure 11 represents the relative likelihood that a utility should consider performing these tasks. These recommendations are based on a combination of trends that NESCOR has seen in the industry and the level of expertise needed to perform these tests. To some degree, this also represents the relative risk target systems represent to the utility, as the compromise of the control servers is generally considered higher risk than the compromise of a single embedded ﬁeld device or its network communications.

The colors in Figure 11 can be interpreted as:

Green: Tasks that should be performed most frequently, require the most basic of penetration testing skill, and can often be performed by internal security teams.
Yellow: Tasks that are commonly performed and require moderate penetration testing skill.
Orange: Tasks that are occasionally performed and may require higher levels of expertise.
Red: Tasks that are infrequently performed and require highly specialized skills not often found inhouse.

The methods introduced here provided a security test basis for the organization in the United States. According to the practice experience, the methods of penetration testing are almost the same. The unequal test results are mainly caused by diﬀerent executor.

6-2-2　Fuzz Testing Method

Fuzz testing was used to be the main skill of the software black box test. It is always vastly used to test the limitation value of the functions in the IT system as well as SCADA and ICS environment. The main purpose of fuzz testing is to ﬁnd the possible fault existing in software as well as the limit violation of the input values. The core skill of fuzz testing is based on the whole input data that do not always comply with the system logic. Fuzz testing does not care or expect which of the data will cause an abnormal situation. It only sends a large number of random numbers to the test target in order to observe the responses.The main usage of fuzz testing is testing the communication protocol security of ICS. The random number data will cause the input ﬁeld to receive an abnormal value. The abnormal response procedure and the stability of the network application system could be tested by means of the input data varies based on a random number. A general fuzz testing procedure is shown in Figure 12.

Figure 12 Fuzz testing procedure

Figure 13 illustrates the whole procedure of fuzz testing test plan.AT the ﬁrst stage, the communication protocol should be studied completely in order to determine the input ﬁeld, data type, and varying rule. The quantity of test sample and test time for each sample should be evaluated for scheduling purpose. The test data are created at the next stage. Meanwhile, the test samples should be inputted or the test sample rules should be formulated. Based on these test samples and rules, the fuzz testing could be carried out subsequently in order to collect the responded status and parameter values of the test target. The test result is then analyzed to ﬁnd the test samples which cause an abnormal situation. Finally, The test report is created.

Figure 13 Stages of fuzz testing

6-3　Platform Architecture

The test platform is composed of a notebook computer with the necessary operating system and testing software. The notebook should be connected to the network with the test target in order to create a valid connection. User commands of the test are given via a web browser interface. The test result could be checked at the test platform.

The conﬁguration of the test environment is shown in Figure 14.

Figure 14 Penetration testing environment

All the devices of the test targets are connected by a network switch. The corresponding internet protocol (IP) addresses are created by network address translation (NAT) of the switch. Windows operating system and Elipspower Ver.4.5 Build 210 SCADA system software are installed on SCADA Client desktop computer. The IEDs have been tested are GE T60, GE F650 and SEL 351 correspondingly. The network switch is Moxa PT-7710.

The test software installed on the test platform includes Hercules SecDevice that is developed by Onward Security. Several security and vulnerability test tools and the packet recording application program are also installed.

Hercules SecDevice is a security assessment tool designed for connected products, and provides automated features from test environment conﬁguration to security assessment. Test targets include web and wireless security. The content covers known and unknown security vulnerabilities, as well as user- friendly design to help users to quickly get started. Most common vulnerabilities listed in OWASP TOP 10 and SANS TOP 25 are also covered. Onward research team provides quick test project updates to ensure coverage of the latest security issues. The testing methods are shown in Table 3.

Table 3 Hercules SecDevice testing methods